Section Highlights¶
- Series Structure , item count , NaNs
- Descriptive Ststistics
- Series Arithmetics
- Sorting Series
- filtering and Custom Transformations
import pandas as pd
Reading in Data with read_csv()¶
CSV : Comma Separated Values (.csv) File
A type of text File containing values delimited by or separated by comma
- A CSV file is simply a text file containing values separated by commas
#Read the CSV by passing the URL
#drinks = pd.read_csv('https://andybek.com/pandas-drinks')
drinks = pd.read_csv('drinks.csv')
drinks.head(3)
#I only want two columns
pd.read_csv('drinks.csv' , usecols = ['country' , 'wine_servings'])
#I want "Country" to act as a Index for the DataFrame
pd.read_csv('drinks.csv' , usecols = ['country' , 'wine_servings'], index_col = 'country')
alcohol = pd.read_csv('drinks.csv' , usecols = ['country' , 'wine_servings'], index_col = 'country')
#The output is still a Dataframe and not a Series
alcohol.head()
type(alcohol)
Note : "pandas.read_csv" method always returns a Data Frame
But there is this "squeeze" method
squeeze : bool, default False : If the parsed data only contains one column then squeeze that DataFrame and return a Series.
alcohol = pd.read_csv('drinks.csv' , usecols = ['country' , 'wine_servings'], index_col = 'country' ,squeeze = True)
alcohol.head()
alcohol = pd.read_csv('drinks.csv' , usecols = ['country'],squeeze = True)
type(alcohol)
alcohol.head()
alcohol = pd.read_csv('drinks.csv' , usecols = ['country' , 'wine_servings'], index_col = 'country' ,squeeze = True)
alcohol.head()
alcohol.shape
#size attribute gives us the length of the series
#We ae looking at a Series of length 193
alcohol.size
alcohol #You can see the "Length" here
Remember : A Series consists of a sequence of values and associated Labels.
#We can access the values :
#That gives us an Array of values
alcohol.values
#Lets look at the associated Index
alcohol.index
alcohol.values.size == alcohol.index.size
Observation
- The size of the values array is equal to the size of the index array
SHAPE attribute¶
"series".shape returns a Tuple¶
Note : Series are only one Dimensional
alcohol.shape
Observation
- We got a Tuple
- We can access the first (or the only element) of this tuple with "alcohol.shape[0]"
alcohol.shape[0]
alcohol.size
alcohol.size == alcohol.shape[0]
or¶
len(alcohol)
Size and Shape¶
.size : number of elements in the series
.shape : tuple of the dimensions for a Series (1D) shape, i.e, length for series
len() : Python built-in function
Unique Values and Series Monotonicity¶
Check if a Series contains a Sequence of Unique values
#Look at the alcohol Series
alcohol.head()
alcohol.is_unique
#Checking uniqueness just for the head()
alcohol.head().is_unique
alcohol.head(15)
alcohol.head(15).is_unique
Observation
- 45.0 is repeated
- "country" is the index , we are not looking at the unique Indexes , we are checking the unique values.
nquique() gives the count of unique values in a Series¶
#For complete DataFrame
drinks.nunique()
alcohol.nunique()
Observation
- nunique does not count the NA
#If you want to include the NA as well
alcohol.nunique(dropna = False)
#For complete DataFrame
drinks.nunique(dropna = False)
The nunique() method returns the number of unique values for each column¶
drinks.shape
#For complete DataFrame
#nunique count does not include NAN by default
drinks.nunique()
#we pass : dropna = False : so NANs are included if any
drinks.nunique(dropna = False)
is_monotonic() Attribute¶
#Notice in the below series , the values are always increasing
pd.Series([1,2,3]).is_monotonic
#Below is a series which is increasing and Stagnating
pd.Series([1,2,3, 3 , 3 , 3 ]).is_monotonic
#Below is a series which is increasing and Stagnating and increasing
pd.Series([1,2,3, 3 , 3 , 3 , 48 , 79 , 100 ]).is_monotonic
#Below is a series which is increasing and Stagnating and increasing and then we add a "7"
#Because at "7" it is a significant decrease , it is no longer Monotonic
pd.Series([1,2,3, 3 , 3 , 3 , 48 , 79 , 100 , 7]).is_monotonic
Observation
- Since the result is False , we can conclude its not a Monotonic Sequence
#Below is a series which is increasing and Stagnating and increasing
pd.Series([1,2,3, 3 , 3 , 3 , 48 , 79 , 100 ]).is_monotonic_increasing
#Below is a series which is increasing and Stagnating and increasing
pd.Series([1,2,3, 3 , 3 , 3 , 48 , 79 , 100 ]).is_monotonic_decreasing
#When we use reversed()
reversed_list = list(reversed([1,2,3, 3 , 3 , 3 , 48 , 79 , 100 ]))
reversed_list
#Now we reverse the List and check for the Monotonicity
pd.Series(reversed([1,2,3, 3 , 3 , 3 , 48 , 79 , 100 ])).is_monotonic
pd.Series(reversed([1,2,3, 3 , 3 , 3 , 48 , 79 , 100 ])).is_monotonic_decreasing
pd.Series(reversed([1,2,3, 3 , 3 , 3 , 48 , 79 , 100 ])).is_monotonic_increasing
The count() Method¶
#Count the values of a Series
alcohol.count()
#Count the values of a DataFrame
drinks.count()
#But the Size of the Series was 193
#Then why is the count() method showing just 162
#What accounts for this difference?
alcohol.size
Note:¶
count() method only looks at values that are not Null
Series.count() : Returns the number of non-NA / null observations in the Series
import numpy as np
s = pd.Series([0.0 , 1.0 , np.nan])
s.count()
Difference between Series.size and Series.count¶
Series.size : Gives a count of all the values
Series.count() : Gives the count of non-NA values
#Lets check if the alcohol Series has Na
alcohol.hasnans
drinks.head()
drinks['beer_servings'].hasnans
drinks['country'].hasnans
#Lets check if the alcohol Series has Na
alcohol.hasnans
Observation So we verified that our Series has NaNs
Accessing and Counting NAs¶
#The Total number of values in the Series is
alcohol.size
#The number of non-Null values in the Series is
alcohol.count()
Now the Question is : How do we look at the NULLs¶
Remember : Nulls or NAs indicate the absence of Data¶
What Countries in our "alcohol" Series do not have Wine Servings?
drinks.head()
alcohol
alcohol.isnull()
drinks = pd.read_csv('drinks.csv')
drinks.head(2)
Accessing and Counting NA's¶
alcohol = pd.read_csv('drinks.csv' , usecols = ['country' , 'wine_servings'] , index_col = 'country' , squeeze = True)
alcohol.head(5)
alcohol.size
alcohol.count()
Nulls or NAa indicate the absence of a value
What countries in our Alcohol Series do not have Wine Servings¶
alcohol.isnull()
Observation
- Afghanistan and Yemen are the two countries (isnull = True) , which do not have Wine Servings
type(alcohol)
#Creating a Boolean Mask
#We get back another Series with only the countries with missing Wine Serving Data are selected
alcohol[alcohol.isnull()]
#Is we just want to look at the country names (isnull = True) or where the values are NAN
#Since Country is the index for alcohol series
#We get a Pandas object
alcohol[alcohol.isnull()].index
type(alcohol[alcohol.isnull()].index)
#Store it in a List
list(alcohol[alcohol.isnull()].index)
#One way to count the Nulls is to find the length of the Series that was produced above
#Simply wrap the above code in a len() function
#But this is not a pythonic way
len(list(alcohol[alcohol.isnull()].index))
#One of the most Pandorable way to get the count of NAN
alcohol.isnull().sum()
#Booleans are integers in Python : True = 1 and False = 0
sum([True , False , True]) #2 Trues and 1 False = 1 + 0 + 1
all = alcohol.size #Total length
nonnulls = alcohol.count() #count() does not include nulls
nulls = alcohol.isnull().sum()
print(all)
print(nonnulls)
print(nulls)
all == nonnulls + nulls
We can use isna in place of isnull¶
Series Accounting¶
.size : Number of elements in the series
- series.size#193
.count() : Number of non null elements
- seris.count() #162
.isna().sum() : number of null elements
- series.isna().sum() #31
BONUS Approach¶
import pandas as pd
import numpy as np
This is another approach to isolate NULLS. A more elegant way.
Sequential or Vectoirised ops¶
Vectorization is the process of going from applying computations to each element (element wise computations) to running them on the entire collection at once.
So, Vectorization refers to the concept of running operations on entire arrays.
When we use Pandas , we are also relying on Numpy , therefore vectorization behind the scenes contributes to the performance gains relative to an approach that is based on regular Python loops, which are most commonly sequential
#unfunc ---> Universal Function
#We will look at np.isnan function
pd.Series(data=[True, False , None, 2] , dtype = float)
ser = pd.Series(data=[True, False , None, 2] , dtype = float)
#We pass this Series to the Numpy isnan() function
#We get back a Series of Booleans indicating whether each of the elements in the Series is null or not
np.isnan(ser)
np.isnan(ser).value_counts()
#Applying isnan() to alcohol Series
alcohol[np.isnan]
drinks.head(2)
np.isnan(drinks.wine_servings).value_counts()
drinks[np.isnan(drinks.wine_servings)]
drinks.columns
drinks[np.isnan(drinks.wine_servings)].shape
drinks[drinks['wine_servings'].isna()].shape
drinks[drinks['wine_servings'].isna()]
OR¶
#To check the length of the resulting Series
alcohol[np.isnan].size
len(alcohol[np.isnan])
alcohol[np.isnan].shape
What if we are interested in everything except nulls¶
notnull() and notna()
notnull() will return a series of Booleans indicating whether each record is null or not¶
But here , this time : "True" values are identified as notnull()
alcohol.notnull()
#We can use this resulting series as a Boolean mask in indexing for nulls only
alcohol.loc[alcohol.notnull()]
Observation¶
- We now see the countries which have the wine servings
drinks[drinks['wine_servings'].notnull()]
#Get the sum of not null
alcohol.notnull().sum()
#Lets check if the count of not null + count of null == the size of the Series
alcohol.notnull().sum() + alcohol.isnull().sum() == alcohol.size
OR
print(drinks.wine_servings.notnull().sum())
print(drinks.wine_servings.isnull().sum())
print(drinks.wine_servings.size)
162 + 31
drinks.wine_servings.notnull().sum() + drinks.wine_servings.isnull().sum() == drinks.wine_servings.size
notnull() is same as notna()¶
drinks.wine_servings.notna().sum() + drinks.wine_servings.isnull().sum() == drinks.wine_servings.size
Booleans are literally numbers in Pythons¶
Bools as Ints¶
- True ----> 1
- False ----> 0
The bool type inherits from (is a subclass of) int
bool --> int --> object
#Example :
True + 19
True + True - False + True * 3
1 + 1 - 0 + 1 * 3
type(True)
type(False)
SKILL Challenge¶
1. Isolate the non-nulls in the alcohol series and assign them to the variable wine_servings.¶
wine_servings = alcohol[alcohol.notna()]
2. What is the Total wine consumed by all the countries in wine_servings?¶
wine_servings.sum()
3. In the wine_servings dataset , what was the total wine consumed by countries that consumed less than 100 servings?¶
Hint : Apply a Boolean mask to identify such countries , than sum()
wine_servings.head(2)
wine_servings[wine_servings < 100].shape
wine_servings[wine_servings > 100].shape
wine_servings[wine_servings < 100].sum()
Dropping and Filling NAs¶
We want to completely exclude NAs from our series
#dropna() simply excludes NAs and returns a new Series
alcohol.dropna()
#One way re-assignment
#alcohol = alcohol.dropna()
#Another way : inplace parameter
#alcohol.dropna(inplace = True)
We can replace the NAs with something more meaningful¶
OR Something that is more analytically useful for us¶
fillna() method¶
In the fillna() method we can specify what values we want to replace with and also pass the inplace parameter
#alcohol.fillna(100 , inplace = True)
alcohol.fillna(100 , inplace = False)
Note : Both .dropna() and .fillna() methods return a copy of the series unless we specify inplace = True
Descriptive Statistics¶
Metrics that allow us to characterise or Describe our Data
What is the Total Wine consumption across all countries in the dataset?¶
#We simply apply the sum() function to the Series itself
#Quick thing to note here is the NAs will be automatically excluded from this calculation
alcohol.sum()
Observation
- We see that more than 8000 servings of wine were consumed in 2010
But in itself , this number is not very meaningful. For example we don't know how this total consumption is distributed. How many countries are contributing to this Total and by how much , on Average ? what the Median is? and so on
#Average wine consumption across all the countries
#To calculate the Average we can make use of two functions
#We combine sum() with count()
#count() will give us the number of items which do not have null values
#By dividing sum by count we get the Average
alcohol.count()
alcohol.sum() / alcohol.count()
OR
alcohol.mean()
MEDIAN¶
#Median is the middle most number
If we have a list containing Odd number of items like above , the median is simply the middle most element
But when we have an even list of numbers , we take the average of the two middlemost numbers
- We sort the list and take the two middle most elements and take the Average
#median
alcohol.median()
Another way of looking at the Median is by ordering our Data and looking at the 50th Quantile . In Pandas we can do this by using the Quantile method.
alcohol.quantile(q = 0.5)
or
alcohol.quantile(0.5)
One thing to note here is :¶
print("The MEAN in the alcohol Series is so much higher than the Median : " + str(alcohol.mean()) + " > " + str(alcohol.median()) )
This indicates that the Distribution of Wine_Servings is Right skewed or Positively skewed.
Positively skewed means , There are countries with large wine_servings that distort our mean but not much change our median.
The mean is the average or the most common value in a collection of numbers.¶
Statistics is the study of how to collect, Organize , Analyze and interpret Numerical information and data.
Statistics is used in HEalthcare and other disciplines too to help aid in Decision making. Understanding statistics is necesary to understand certain processes in healthcare . (To figure out what to do ? not How we do?)
Entire Population = Census¶
Descriptive Statistics invlove methods of organizing , picturing and Summarizing information from Samples and Populations.
Inferential Statistics invloves methods of using information from a Sample to draw conclusions regarding the Population.Therefore Inferential Statistics can only be done ON a Sample. We are trying to infer from the Sample.
Variables are of two types : Qualitative and Quantitative
Quantitative variables : Interval and Ratio
Basic Guidelines for planning a Statisctical Study:¶
- State a Hypothesis
- Identify the Individuals of interest
(Individuals could be anything : hospitals , people , students , banks , Products)
You have to measure the variables about these indivisuals . So, Step 3 is to "Specify all of the Variables" you will need to measure about these individuals (ofcourse they relate to the Hypothesis)
STep 4 is to determine whether you want to use the entire population in your study or just the Sample.
- If you choose a sample , choose a Sampling method.
Now that you figured out your Hypothesis , you got your individuals , you got your variables. You figured out whether you are going to use a sample or a population. You selected your sampling method. Now the Step 5 is to "Address the ethical concerns before Data Collection" . Asking some sesitive Questions, you think about privacy. So, you have to sit down and think about these Ethical concerns.
Now its time for data Collection.
Remember: stating a Hypothesis is the most important part. without that , you will not even know what data to collect.
You either use Descriptive or Inferential Statistics to answer your Hypothesis.
Note any concerns about your Data Collection or Analysis. Make recommendations for future studies.
Example : STEPS 1 to 3
Sampling Ethics and Data Collection¶
- Either Collect Data or use existing Dataset
- Can use a government Dataset for Population mesasures
Can collect data from a Sample for Estimates¶
- Need to choose Sampling approach
- Will need consent if legally found to be "Human Research"
- May need consent from parents to collect data about Children
Population is also called a Census¶
- In a Census , measurements or Observations from the entire Population are used.
- In a Sample , measurements or Observations from part of the Population are used.
Because if you pick a Census , you will do a different kind of Analysis and if you pick a sample you will do a different kind of Analysis
Frequency Histogram¶
Frequency Histogram is a Specific type of Bar chart made from data in a Frequency table.
Both Frequency Histograms and Relative Frequency Histograms are type of Bar Charts.
Histograms reveal the Distribution of Data.
This is how a Frequency Table looks like. Here we have added "RElative Frequency" column as well
Relative Frequency is nothing but the Proportion or percentage:
But ultimately you will get the same pattern in your Histogram , whether you use Frequency or Relative Frequency
What is a Distribution ?¶
- Distribution is actually just the SHAPE
- It is the shape that is made if you draw a line along the edges of a Histogram's Bars.
5 Main Types of Distributions¶
- Normal Distribution (Also called mound-shaped symmetrical)
- Uniform Distribution
- Skewed Left Distribution
- Skewed Right Distribution
- Bimodal Distribution
Normal Distribution:
Uniform Distribution
Skewed Left Disttibution
Skewed Right Distribution
Bimodal Distribution
Cumulative Frequency¶
Its like keep accumulating things :
BAR Graph : A Bar graph can display Quantitative or Qualitative Data¶
- Bars can be vertical or Horizontal
- The Frequency Histogram and Relative Frequency Histogram are Special cases of a Bar Graph:
- When you are comparing bar charts (before and After) make sure you use the same scale
PARETO Chart¶
Pareto chart is a different kind of Special bar graph
- Here the height of the Bar indcates the Frequency of an Event.
- Example : Look at the tallest bar "Damaged Radar Core" happened 31 times.
- Pareto charts are not used much in Healthcare
- This is a Qualitative PIE chart
- This is a Quantitative PIE chart
Remember : "FREQUENCY" is always about Quantitative Data
Mesaures of Central Tendency¶
- Mode
- Median
- Mean
Left looks more centery and right is Skewed
MODE :¶
- Mode is the number in the dataset that occurs Most Frequently
- It is possible to have No Mode
A lot of Datasets have no mode. There is just no repeated value in them.
import numpy as np
import pandas as pd
test_mode = pd.Series( [1,2,3,4,5])
test_mode.mode()
Is is also possible to have more than one MODE
test_mode = pd.Series( [1,2,2,3,4,5,3,4,5])
test_mode.mode()
What does the MODE tell you?
- Not much
- The most "Popular" answer
- The Most common Result
- Not used a lot in healthcare
- So, the MODE is not a Robust measure (Just if one value is added or removed , the Mode may change)
- Specially in healthcare
MEDIAN¶
- Its the Middle of the Data
- MEDIAN is the centre of the Data
- Every set of Quantitative Data can be sorted in order of Lowest to highest
- Sometimes there are Repeats in the Data
- Sometimes there are Outliers
- Sometimes all the Data values are almost the same
- Even So, they can be arranged in order
- The Median is the number that is halfway
- So, when you take the values , put them in order and count halfway up , thats what the Median is going to be.
How to find the MEDIAN¶
- Order the Data from smallest to largest
- 42 , 33, 21 , 78 ,62
median_test = [42 , 33, 21 , 78 ,62]
sorted_median_test_ODD = sorted(median_test)
#This is the odd version
sorted_median_test_ODD
median_test = [12 , 7 , 3, 8 ,1 ,9]
sorted_median_test_EVEN = sorted(median_test)
sorted_median_test_EVEN
#We have ordered the Data from smallest to largest in both the lists
#sorted_median_test_EVEN and sorted_median_test_ODD
If there is an ODD number of values, the MEDIAN is the Middle data value¶
sorted_median_test_ODD
ODD = pd.Series(sorted_median_test_ODD)
ODD.median()
If there is an Even number of values , the MEDIAN is the Sum of the middle two values divided by 2¶
sorted_median_test_EVEN
EVEN = pd.Series(sorted_median_test_EVEN)
EVEN.median()
But what if you have a hundered and fifty numbers?¶
- You can still arrange them in order
- After arranging the values in Order, you have an "Ordered Set"
What does the MEDIAN tell you ?¶
- The 50th Percentile of the Data¶
What it means is , 50% or half of the Data Points are below the MEDIAN and the other half are above.
And so, its also known as the Middle Rank of the Data
Whats nice about the MEDIAN is, it doesn't really care about the ends of the Data. And OUTLIERS don't really bother it. This way MEDIAN is very RESISTANT . Its very STABLE.
MEAN , TRIMMED MEAN and WEIGHTED AVERAGE¶
- MEAN is not resistant to Outliers
- MEAN is not very stable (If there is a higher value , it will change drastically)
- What can you do to try and make the MEAN more stable?
- We can trin data off from each end, so the Outliers get Cut off.
- This way , you might lose some data
- Lets look at another way of doing this :
- Lets say you have 100 Data points
- And you want to trim 5% of the data from top ans 5% from the bottom
- 5% of 100% is just 5 data points
- So, you go ahead and put the data in order(sort)
- And cut off 5 each from top and bottom
- So, you got 90 values left in the middle
- Now you make a MEAN out of them
- And thats a 5% trim
- And then you have to tell the people that - Here's the original MEAN and here's the 5% trimmed MEAN.
- This might give you a stable estimate of the MEAN
Whats happening in the Right Skewed Distribution?¶
- The MEAN is getting dragged around by that TAIL
- You can see that the MEAN is at the RIGHT side of the MEDIAN
Here the MEDIAN is more resistant , its sort of hanging on to the bottom of the Data. But the Right TAIL is pulling the MEAN up and the MODE is the lowest one.
- So if i get this Printout where MEAN is towards the right and MEDIAN is in the centre
- Meaning MEAN is the highest , we can say even without looking at the Histogram, this is RIGHT Skewed
LEFT Skewed Distribution:
- Here the tail is dragging the MEAN down , notice the MEDIAN is more resistant , does not get dragged as much
And the MODE stays at the higher part of the Data where there is more Data
- So if i get a printout and notice the MEAN is the lowest , MEDIAN is in the middle and the MODE is the highest
- I will understand this is LEFT Skewed
EDA : Exploratory Data Analysis is the first step in Data analysis¶
For categorical vaiables , we can look at the MODE
Webscraping Data using Pandas¶
import pandas as pd
#Retrieve HTML table Data
url = 'https://www.basketball-reference.com/leagues/NBA_2019_per_game.html'
html = pd.read_html(url , header = 0) #0 row is the header
df2019 = html[0]
html
df2019
df2019[df2019.Age == 'Age']
df2019[df2019.Age == 'Age'].index #these are the indexes from which we have to drop
#Dropping all the headers that are Redundant
#Data Cleaning
raw = df2019.drop(df2019[df2019.Age == 'Age'].index)
raw.head()
#Data Dimension
raw.shape
#Check for missing values
raw.isnull().sum().head(4)
#Filling all the missing values with 0
df = raw.fillna(0)
#Write to CSV File
df.to_csv('nba2019.csv' , index = False)
#Type ls to check if the File has been created
%ls
#We read the Data back in
df = pd.read_csv('nba2019.csv')
df.shape
df.shape[0]
#We want no beaks between , we wamt to see all of the rows
pd.set_option('display.max_rows' , df.shape[0] + 1)
df
#Reverting back to the default
pd.set_option('display.max_rows' , 10)
df
#Overview of Datatypes of each column in the Dataframe
df.dtypes.head(5)
#Show specific Data types in our DataFrame
df.select_dtypes(include = ['number'])
#Show specific Data types in our DataFrame
df.select_dtypes(include = ['object'])
#df.select_dtypes(include=['datetime64'])
Which player scored the most points per game ?¶
df.PTS.max()
#Filter : df ---> PTS == max()
#This is Conditional Selection
df[df.PTS == df.PTS.max()]
What Team is the Player from ?¶
PlayerMaxPoints = df[df.PTS == df.PTS.max()]
PlayerMaxPoints.Tm
which Position is the Player Playing as?¶
PlayerMaxPoints
PlayerMaxPoints.Pos
How many games did the Player played in the Season?¶
PlayerMaxPoints.G
Which Player Scored more than 20 points per game?¶
df[df['PTS']>20]
Which player had the highest 3Point Field Goals per game?¶
df[df['3P'] == df['3P'].max()]
Which player had the highest Assists per game?¶
df[df['AST'] == df['AST'].max()]
Group By() function¶
Which player scored the higest (PTS) points from the Los Angeles Lakers?¶
#The below code groups the data by the TEAM >>> df.groupby('Tm')
#Then it will select the specific team that we want , that is LAL
LAL1 = df[df['Tm'] == 'LAL']
LAL1.shape
LAL.shape
LAL = df.groupby('Tm').get_group('LAL')
LAL
df.head()
OKC = df.groupby('Tm').get_group('OKC')
OKC.head()
Of the 5 Positions , which position scores the most points ?¶
#We first group by the Position
df.groupby('Pos').PTS.describe()
Observation
- Some positiopns are hybrid , eg: C-PF (Both as a Centre and a Power Forward)
- Meaning , some players played two positions
- So, the Hybrid positions count is low, so we will remove the Hybrid positions
- (C-PF , PF-SF , SF-SG , SG-PF , SG-SF )
#First we will define a Variable called "Position"
#We will make a List of positions that we want to be shown
position = ['C' , 'PF' , 'SF' , 'PG' , 'SG']
POS = df[df['Pos'].isin(position)]
POS.head(5)
POS.Pos.unique()
#Now lets look at the Descriptive Statistics
#We are viewing Descritive statistics of only one column
POS.groupby('Pos').PTS.describe()
Histograms¶
We will also try to answer this Question by showing some Hisogram Plots. So, to make it a bit easier lets create a Subset DataFrame.
#Here we are going to select the columns 'Pos' and 'PTS' ---> PTS= df[['Pos', 'PTS']]
#Then we have a list of only the Five positions we are interested in
#position = ['C' , 'PF' , 'SF' , 'PG' , 'SG']
PTS = df[['Pos' , 'PTS']]
position = ['C' , 'PF' , 'SF' , 'PG' , 'SG']
PTS = PTS[PTS['Pos'].isin(position)]
PTS
Pandas Built-in Visualization¶
PTS.head(2)
PTS['PTS'].hist(by = PTS['Pos'])
Observation
- We have made Several Histogram Sub-Plots according to the PTS['Pos'] column
#Letsa change the Layout
#This will show you one Row and Five columns
#We customize the fig size as well
PTS['PTS'].hist(by = PTS['Pos'] , layout =(1,5), figsize=(16,2))
#You can further go ahead and Customize the Bins
#We can do this using Seaborn
import seaborn as sns
import matplotlib.pyplot as plt
g = sns.FacetGrid(PTS , col = 'Pos')
g.map(plt.hist, "PTS");
PTS['PTS'].describe()
Histograms¶
Histogram¶
Example:¶
Here are the Scores of 15 Students
scores = [88 , 48, 60, 51, 57 , 85 , 69, 75, 97 , 72, 71 , 79, 65, 63 , 73]
len(scores)
scores = pd.Series(scores)
scores
sorted(scores)
scores.describe()
scores.median()
sorted(scores)
scores.hist()
scores = [88 , 48, 60, 51, 57 , 85 , 69, 75, 97 , 72, 71 , 79, 65, 63 , 73]
scores = scores * 2
scores = pd.Series(scores)
scores.hist()
BACK TO UDEMY BOOTCAMP¶
drinks = pd.read_csv('drinks.csv')
drinks.head(2)
import pandas as pd
import numpy as np
alcohol = pd.read_csv('drinks.csv' , usecols = ['country' , 'wine_servings'], index_col = 'country')
alcohol.mean()
alcohol.median()
alcohol.hist()
len(alcohol)
alcohol.describe()
The mean in math and statistics summarizes an entire dataset with a single number representing the data’s center point or typical value. It is also known as the arithmetic mean, and it is the most common measure of central tendency. It is frequently called the “average.”
alcohol.mean()
alcohol.median()
alcohol.hist()
Observation
- The MEAN in the Alcohol series is so much higher than the MEDIAN
- Mean is 50.7 and Median is 11.5
- This indicates that the Distribution of wine_servings is Right Skewed or Positively Skewed
- In other words , there are countries with large Servings that Distort our MEAN
X-Axis : Will have values increasing from left to right
Y-Axis : will have frequencies or how many times those values show up in the distribution
Positive Skew :¶
- Whenever we say say that it is postively skewed , we mean that the Distribution of values extends into the Right hand tail. We have more Data on the Right hand side of the Distribution and more extreme Observations. There are more number of high value obsevations than there are low value observations
scores = [1,1,1,2,2,2,2,4,4,4,4,4,5,5,5,5,5,5,6,7,9,10,11,12,12,12,13,14,15,
17,19,20,21,23,25,33,35,40,44,45,46,47,48]
scores = pd.Series(scores)
scores.mean()
scores.median()
scores.hist()
Negative Skew:¶
- Negative Skew is exactly the opposite of Positive Skew
- The Distribution of values extends to the Left hand side of the tail
- We have more data on the Left hand side of the Distribution and more extreme observations
- There are more number of LOW value observations than there are high value observations
scores = [0,0.8,1,1.2,1.5,2,2.2,2.5,3,3.2,3.5,3.6,3.7,4,4.1,4.2,4.4,4.5,5,6]
scores = pd.Series(scores)
scores.mean()
scores.median()
scores.hist()
alcohol = pd.read_csv('drinks.csv' , usecols = ['country' , 'wine_servings'], index_col = 'country')
alcohol.quantile(0.5)
alcohol.hist()
Observation
- This is a very positively skewed distribution
- Most of our observations are concentrated on the lower end
- And then we have values that extend into the far right
- Whenever this is the case, the MEAN will exceed the MEDIAN
- Because it will be pulled by the extreme Observations
Remember : As we said MEDIAN is the 50th Quantile
IQR : Inter Quartile Range¶
IQR is the difference between the First and Third Quartiles
alcohol.median()
#IQR is calculated as:
iqr = alcohol.quantile(0.75) - alcohol.quantile(0.25)
iqr
Observation
- So, our alcohol series has an InterQuartile Range of 58.25
- Which is quite a significant Difference
- This number as well confirms that our Data is not distributed Symmetrically
- The MEDIAN was 11.5
- The IQR : 58.25 is five times the Median
The InterQuartile Range measures the spread of the middle half of your data. It is the range for the middle 50% of your Sample. Use the IQR to assess the variability where most of your values lie.
Larger values indicate that the central portion of your data spread out further. conversely smaller values show that , the middle values cluster more tightly. It is also helpful to test whether your data is normally distributed.
InterQuartile Range is one of the serveral measures of variability:
To visualize the interquartile range, imagine dividing your data into quarters. Statisticians refer to these quarters as quartiles and label them from low to high as Q1, Q2, Q3, and Q4. The lowest quartile (Q1) covers the smallest quarter of values in your dataset. The upper quartile (Q4) comprises the highest quarter of values. The interquartile range is the middle half of the data that lies between the upper and lower quartiles. In other words, the interquartile range includes the 50% of data points that are above Q1 and below Q4.
IQR is less affected by Outliers. typically , use the IQR with a measure of Central Tendency , such as the MEDIAN to understand your Data's Center and Spread. This combination creates a fuller picture of your data's distribution.
Unlike the more familiar MEAN and Standard deviation, the IQR and Median are Robust measures. They are not influenced by the Outliers because they dont depend on every value. Additionally , like the MEDIAN , IQR is superb for skewed Distributions. For Normal Distributions , you can use the standard Deviation to determine the percentage of obsrervations that fall specific distances from the MEAN. For Skewed distributions , IQR is an excellent alternative.
IQR = Q3 - Q1¶
IQR takes the Third Quartile Value and subtracts the First Quartile value. Equivalently , the InterQuartile Range is the region between the 75th and 25th Percentile (75 - 25 = 50% of the Data)
Using the IQR formula we need to find the values for Q3 and Q1. To do this, order your data from Low to high and split it into four Equal Proportions.
Q3 = alcohol.quantile(0.75)
Q1 = alcohol.quantile(0.25)
print(Q3)
print(Q1)
iqr = alcohol.quantile(0.75) - alcohol.quantile(0.25)
iqr
The Box in the Box plot is your InterQuartile Range. It contains 50% of your Data. By comparing the size of these boxes , you can understand your data's variability. More dispersed distributions have wider boxes.
Additionally , Find where the MEDIAN line falls within each InterQuartile box.If the MEDIAN is close to one side or the other of the Box , its a skewed distribution. When the MEDIAN is near the centre of the Interquartile range , your distribution is symmetric.
You can also find Outliers using IQR . Since IQR is robust and not influenced by Outliers:
Back to Bootcamp lecture¶
min and max¶
alcohol.min()
alcohol.max()
Observation¶
- alcohol dot min gives us the minimum Wine servings in our alcohol series
- And alcohol dot max gives us the maximum
Standard Deviation¶
alcohol.std()
Standard Deviation is just the Square Root of the Variance
VARIANCE¶
alcohol.var()
Square Root of Standard Deviation is equal to Variance¶
alcohol.std()**2 == alcohol.var()
76.134917 ** 2
The describe() method¶
Almost everything that we previously calculated manually , could be obtained using this single method.
alcohol.describe()
This method returns a Pandas Series containing Descriptive Statistics and it is very convenient and Quick.
- It gives us a really good numerical sense of what we are working with.
- We can customise this a bit
#We can specify we also need the 79th and the 19th Percentiles as well
alcohol.describe(percentiles = [0.79 , 0.19])
Observation
- The Non-Null count of wine_servings is 162
- The Average value is 50.7 with a Standard Deviation of 76.1%
- The minimum wine_serving for a country is 1
- The maximum wine_servings are 339
- Median value is 11.5 (By comparing the median to the mean, you can get an idea of the distribution of a dataset. When the mean and the median are the same, the dataset is more or less evenly distributed from the lowest to highest values. When the mean and the median are different then it is likely the data is not symmetrical but is either skewed to the left or the right.)
median is the data value in the data, above and below which there is an equal number of data points.there is an equal probability of falling above or below it.
Mean, median, and mode are three kinds of "averages". There are many "averages" in statistics, but these are, I think, the three most common, and are certainly the three you are most likely to encounter in your pre-statistics courses, if the topic comes up at all.
The "mean" is the "average" you're used to, where you add up all the numbers and then divide by the number of numbers. The "median" is the "middle" value in the list of numbers. To find the median, your numbers have to be listed in numerical order from smallest to largest, so you may have to rewrite your list before you can find the median. The "mode" is the value that occurs most often. If no number in the list is repeated, then there is no mode for the list.
Mode- It is the value which has highest frequency in population.
The median, just like the mean, is a single-number indicator of how a particular distribution is centered. When the distribution is symmetrical, the two numbers coincide. But when the distribution is skew (e.g. in income statistics) the median is a better indicator (gives a more realistic picture at a glance).
Median is the most robust estimator to outliers. Outlier is one huge problem in statistics since it happen all the time. When people record the data wrongly, when there are strange cases in the world (income). So median is good idea to look when that happens.
They are simple descriptive statistics - so 3 simple numbers tell you a fair amount about your data and you can use them in your reporting.
But don’t underestimate the value of plotting/graphing your data. This will give you pretty much the same information as those 3 statistics, and even more detail, e.g. it will tell you the range, give a visual indication of the quartiles and the presence/absence of outliers.
Back to the Bootcamp¶
alcohol.describe()
#We can use the include and exclude parameter
#Filter data by type , include float data type and exclude object data type
alcohol.describe(include =float , exclude = object)
since "alcohol" dtype is just "float" , the above output did not change when we specified the include and exclude explicitly
type(alcohol)
mode() and value_counts()¶
mode() gives us the most common item or the value that occurs most frequently in a collection of values
So, the value that appears most number of times or the one that has the highest frequency in the series becomes the mode. Mode is the PEAK of the distribution.
Median is the middle most observation. In other words, its the point in our dataset that separates the dataset into two equal sized parts. And the MEAN is simply the Average value , which we calculate as the sum of all observations divided by the count of all observations.
#Calculate the mode
alcohol.mode()
Observation
- The most frequent occuring wine serving across all countries in our alcohol dataset is 1.
#Lets say we are interested in figuring out how many times this value of 1 occurs.
#To answer this we use a basic Boolean Mask
#we use "==" to compare the integer value "1" to each value in the series
alcohol == 1
#Now we can use this boolean mask to index the alcohol series
alcohol[alcohol['wine_servings'] == 1]
alcohol[alcohol['wine_servings'] == 1].size
Observation
- We know that there are 28 Countries with a wine Serving of 1
value_counts()¶
Calculates the number of occurances for each unique value all in one go
alcohol['wine_servings'].value_counts()
Oservation
- We get back a sorted Series which contains values in the Index and the associated frequencies
#The first value is oocuring how many times
alcohol['wine_servings'].value_counts().iloc[0]
alcohol['wine_servings'].value_counts().values
len(alcohol['wine_servings'].value_counts())
alcohol['wine_servings'].value_counts().tail()
Access the Index values from value counts¶
alcohol['wine_servings'].value_counts()
vc = pd.DataFrame(alcohol['wine_servings'].value_counts())
vc.index
#This is the most frequent value or the MODE
vc.index[0]
len(vc.index)
#This is the least frequent value
vc.index[70]
OR¶
vc.index[len(vc.index)-1]
value_counts() by default :¶
- sort = True
- ascending = False
- dropna = True
- normalize = False
What is normalize doing?¶
PANDAS : value_counts()¶
By default "normalize" is set to "False", but when we set to True , we get the relative frequencies of the unique values in the series .
alcohol['wine_servings'].value_counts(normalize = True)
Observation
- We dont get absolute counts but we get the Relative Frequencies.
- The first record 1 is associated with 17.28 %
- This means that the value 1 accounts for 17.3% of all the unique values in the Series.
idxmax() and idxmin()¶
Which country had the most wine servings in 2010?¶
alcohol.max()
But which country this is ?
#We can set a Boolean Mask
alcohol[alcohol['wine_servings'] == alcohol['wine_servings'].max()]
alcohol['wine_servings'].hist()
#But what we get here is a Series and not the country name separately
alcohol[alcohol['wine_servings'] == alcohol['wine_servings'].max()]
type(alcohol[alcohol['wine_servings'] == alcohol['wine_servings'].max()])
#we can get the country name by isolating the "Index" component of this series
alcohol[alcohol['wine_servings'] == alcohol['wine_servings'].max()]
drinks.head(2)
alcohol = pd.Series(drinks['wine_servings'].values, index=drinks['country'])
alcohol
#we can get the country name by isolating the "Index" component of this series
#This returns a Pandas index object
alcohol[alcohol == alcohol.max()].index
#To get the country name
alcohol[alcohol == alcohol.max()].index[0]
The above procedure is very confusing. The same thing can be done using the idx method.
Get the Index value associated with the MAX value in the Series¶
alcohol.idxmax()
#This is the Series
alcohol.max()
#This is the Dataframe
drinks['wine_servings'].max()
alcohol[alcohol == alcohol.max()]
drinks['wine_servings'].idxmax()
drinks['wine_servings'].iloc[136]
drinks.iloc[drinks['wine_servings'].idxmax()]
drinks.loc[drinks['wine_servings'] == drinks['wine_servings'].max()]
alcohol.idxmin()
alcohol.min()
Important to note :¶
we have seen in the value_counts , there are 28 countries with a min value of 1
alcohol.value_counts().head(1)
But what we see in "alcohol.idxmin" is only one of those countries
alcohol.idxmin()
#This is the actual result
alcohol[alcohol == alcohol.min()]
Conclusion :¶
idxmin() and idxmax() are very convenient methods that will return only the First label associated with the max and min values repectively
alcohol.idxmax()
#we can simply enter this index value to get the row
alcohol['Portugal']
drinks.head(2)
drinks.iloc[0]
⚫⚫🛑➖⚫🛑⚫⚫➖⚫⚫🛑➖⚫🛑⚫⚫⚫⚫🛑➖⚫🛑⚫⚫⚫⚫🛑➖⚫🛑⚫⚫⚫⚫🛑➖⚫🛑⚫⚫⚫⚫🛑➖⚫🛑⚫⚫